Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor(error)!: handle errors more gracefully in rustic_core. #202

Closed
wants to merge 58 commits into from

Conversation

simonsan
Copy link
Contributor

@simonsan simonsan commented Mar 13, 2024

Till now, rustic_core had a lot of unwraps and expects sprinkled over the code base. This made rustic_core and dependents panic in case of errors. This is unacceptable for a library, hence we fixed it now.

Fixes #140

TODO:

  • Implement unit tests for test coverage of some changes
  • Fix integration test
  • 0 (*nix) errors left (due to unwrap/expect being denied)

Possible Panics left over/reintroduced:

@simonsan simonsan added A-errors Area: error handling needs improvement C-enhancement Category: New feature or request C-refactor Category: Refactoring of already existing code labels Mar 13, 2024
@simonsan simonsan marked this pull request as draft March 13, 2024 10:21
Copy link

codecov bot commented Mar 13, 2024

Codecov Report

Attention: Patch coverage is 26.00000% with 444 lines in your changes missing coverage. Please review.

Project coverage is 27.5%. Comparing base (fb291dc) to head (f1218c8).
Report is 106 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/commands/restore.rs 0.0% 123 Missing ⚠️
crates/core/src/blob/packer.rs 54.4% 61 Missing ⚠️
crates/core/src/commands/prune.rs 4.3% 44 Missing ⚠️
crates/core/src/vfs.rs 0.0% 29 Missing ⚠️
crates/core/src/backend/local_destination.rs 15.1% 28 Missing ⚠️
crates/core/src/backend/cache.rs 0.0% 24 Missing ⚠️
crates/core/src/blob/tree.rs 32.1% 19 Missing ⚠️
crates/core/src/commands/copy.rs 0.0% 17 Missing ⚠️
crates/core/src/commands/forget.rs 0.0% 15 Missing ⚠️
crates/core/src/archiver.rs 45.8% 13 Missing ⚠️
... and 21 more
Additional details and impacted files
Files with missing lines Coverage Δ
crates/backend/src/choose.rs 0.0% <ø> (ø)
crates/backend/src/opendal.rs 0.0% <ø> (ø)
crates/backend/src/rclone.rs 87.5% <ø> (+9.7%) ⬆️
crates/core/src/archiver/file_archiver.rs 77.0% <ø> (ø)
crates/core/src/cdc/polynom.rs 75.0% <ø> (ø)
crates/core/src/cdc/rolling_hash.rs 54.3% <ø> (ø)
crates/core/src/chunker.rs 58.6% <ø> (+0.7%) ⬆️
crates/core/src/crypto/aespoly1305.rs 77.7% <ø> (-5.6%) ⬇️
crates/core/src/error.rs 25.0% <ø> (+10.7%) ⬆️
crates/core/src/index/binarysorted.rs 54.5% <ø> (-0.9%) ⬇️
... and 36 more

... and 7 files with indirect coverage changes

@simonsan simonsan marked this pull request as ready for review March 14, 2024 14:41
@simonsan simonsan requested a review from aawsome March 14, 2024 14:41
Copy link
Contributor Author

@simonsan simonsan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some notes where help would be appreciated

crates/core/src/backend/local_destination.rs Outdated Show resolved Hide resolved
crates/core/src/blob/packer.rs Outdated Show resolved Hide resolved
crates/core/src/commands/backup.rs Outdated Show resolved Hide resolved
Comment on lines +122 to +136
// REVIEW: Behaviour could have changed here, check!
index_be
.into_index()
.into_index()?
.into_iter()
.par_bridge()
.for_each(|pack| {
.map(|pack| -> RusticResult<()> {
let id = pack.id;
let data = be.read_full(FileType::Pack, &id).unwrap();
match check_pack(be, pack, data, &p) {
Ok(()) => {}
Err(err) => error!("Error reading pack {id} : {err}",),
}
});
let data = be
.read_full(FileType::Pack, &id)
.map_err(RusticErrorKind::Backend)?;

check_pack(be, pack, data, &p)
})
.collect::<RusticResult<()>>()?;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

due to replacing for_each with map (to be able to use results) behaviour could have changed, check again

crates/core/src/commands/forget.rs Show resolved Hide resolved
crates/core/src/id.rs Outdated Show resolved Hide resolved
crates/core/src/vfs.rs Outdated Show resolved Hide resolved
@simonsan
Copy link
Contributor Author

The CI fails for Test Coverage, I don't intend to write tests at this stage for the changes as I think we should focus on the other tests first, that we wanted to implement (e.g. path related stuff).

I don't want to implement these tests in this PR though, as it's already quite big.

Comment on lines 1140 to 1152
#[rstest]
#[case(0.5f32, false, "size is too small, should be 'false'")]
#[case(1.1f32, true, "size is ok, should be 'true'")]
#[case(1_000_000.0f32, false, "size is too huge: should be 'false'")]
#[allow(clippy::cast_possible_truncation)]
fn test_compute_pack_size_ok_passes(
pack_sizer: PackSizer,
#[case] input: f32,
#[case] expected: bool,
#[case] comment: &str,
) -> RusticResult<()> {
let size_limit = pack_sizer.pack_size()? * 30 / 100;

let size = (input * size_limit as f32) as u32;

assert_eq!(pack_sizer.size_ok(size)?, expected, "{comment}");

Ok(())
}
Copy link
Contributor Author

@simonsan simonsan Mar 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you think of a better way to test the PackSizer::size_ok() + PackSizer::add_size()?

@simonsan
Copy link
Contributor Author

OK, added some more testing, so the coverage test also runs through. 👍🏽

crates/core/src/id.rs Outdated Show resolved Hide resolved
crates/backend/src/choose.rs Outdated Show resolved Hide resolved
crates/backend/src/opendal.rs Outdated Show resolved Hide resolved
crates/backend/src/local.rs Outdated Show resolved Hide resolved
crates/backend/src/rclone.rs Outdated Show resolved Hide resolved
crates/backend/src/rclone.rs Outdated Show resolved Hide resolved
Comment on lines 26 to 51
type Error;

/// Check a response for an error and treat them as permanent or transient
fn check_error(self) -> Result<Self, Self::Error>
where
Self: Sized;
}

impl CheckError for Response {
type Error = backoff::Error<reqwest::Error>;

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this improve?

Copy link
Contributor Author

@simonsan simonsan Mar 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The latest version improves the name (imho) and this makes it possible to use it further down the line with other types. Also, people can use it easier for their own implementation, e.g. in case they want to use another library at a certain point in time handling the error of a response and backing off. So I like the idea of having a more generalized trait for that.

Copy link
Contributor Author

@simonsan simonsan Mar 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/// `ValidateResponse` to add user-defined method `validate` on a response
///
/// This trait is used to add a method `validate` on a response to check for errors
/// and treat them as permanent or transient based on the status code of the response.
///
/// It returns a result with the response if it is not an error, otherwise it returns
/// the associated error.
pub trait ValidateResponse {
    /// The error type that is returned if the response is an error
    type Error;

    /// Check a response for an error and treat it as permanent or transient
    ///
    /// # Errors
    ///
    /// If the response is an error, it will return an error of type [`Self::Error`]
    ///
    /// # Returns
    ///
    /// The response if it is not an error or an error of type [`Self::Error`] if it is an error
    fn validate(self) -> Result<Self, Self::Error>
    where
        Self: Sized;
}

crates/core/src/archiver.rs Show resolved Hide resolved
crates/core/src/archiver.rs Outdated Show resolved Hide resolved
crates/core/src/backend/cache.rs Outdated Show resolved Hide resolved
@@ -296,3 +332,57 @@ impl Parent {
Ok(result)
}
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

internal remark: Review until here.

Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
…r the code base, also handle cli changes with alias

Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
…leanup, add cfg_if for tests and os dependent stuff

Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
Signed-off-by: simonsan <[email protected]>
@simonsan simonsan changed the title feat(error)!: handle errors more gracefully in rustic_core. refactor(error)!: handle errors more gracefully in rustic_core. May 5, 2024
@simonsan simonsan marked this pull request as draft September 30, 2024 16:51
@simonsan simonsan closed this Oct 7, 2024
@simonsan simonsan added the C-zombie Category: A pull request/issue that may be of value at a later stage. label Oct 7, 2024
@simonsan
Copy link
Contributor Author

simonsan commented Oct 7, 2024

Closing this as it's heavily outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-errors Area: error handling needs improvement C-enhancement Category: New feature or request C-refactor Category: Refactoring of already existing code C-zombie Category: A pull request/issue that may be of value at a later stage.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Remove unnecessary unwrap/expect from codebase and improve error handling
2 participants